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Abstract 

Education practice often requires teachers and other school personnel to adapt interventions 
over time in order to address between-student heterogeneity in response to intervention (e.g., 
what works for one student may not work for the other) or within-student heterogeneity (e.g., 
what works now may not work in the future for the same student). An adaptive intervention 
allows education practitioners to do this in a prespecified, systematic, and replicable way 
through a sequence of decision rules that guides whether, how, and when to modify 
interventions. In an adaptive intervention, the practitioner modifies the dosage or type of 
intervention, or the mode of delivery to meet the unique and changing needs of students as 
they progress over time. The sequential, multiple assignment, randomized trial (SMART) is one 
type of multistage, experimental design that can help education researchers build high-quality 
adaptive interventions. Despite the critical role adaptive interventions can play in various 
domains of education, research about adaptive interventions and about the use of SMART 
designs to develop effective adaptive interventions in education is in its infancy. This paper 
defines an adaptive intervention and reviews the components of this design, discusses the key 
features of the SMART, and introduces common research questions for which SMARTs may 

be appropriate. 


Introduction 

Educational practice often requires adapting and readapting an intervention provided to a student 
in order to address the student’s changing needs. Here, adaptation refers to modifying the dosage 
(duration, frequency, or amount), type, or delivery modality of interventions based on ongoing 
information about the student’s progress and response. This approach is often necessary when the 
type of intervention that works for one student may not work for the other (between-student 
heterogeneity), or when the type of intervention that works now for one student may not work in 
the future for the same student, or vice versa (within-student heterogeneity). 


An adaptive intervention is a prespecified, replicable sequence of decision rules that guides 
whether, how, when, and which measures to use to make critical decisions about interventions 
in education settings. Adaptive interventions capitalize on differences in how students respond 
to intervention by providing appropriate modifications for those who need them (e.g., those 
who are not showing the anticipated improvement) and when they need them (e.g., as soon as 
it becomes clear that the intervention is not working adequately). 


As an example, consider a student-level adaptive intervention designed to improve the school 
performance of children with attention-deficit/hyperactivity disorder (ADHD) (Figure 1). This is 
a simplified example taken from a study, Adaptive Treatments for Children with ADHD, 
described in more detail below (Grant # R324B060045 hitps://ies.ed.gov/funding/grantsearch 
details.asp?ID=396; Pl: Pelham; Pelham et al., 2016), which was funded by the U.S. Department 
of Education Institute of Education Sciences (IES). This example includes two types of 
interventions for ADHD: medication and behavioral intervention. At the beginning of the school 
year, all children receive a low-intensity behavioral intervention consisting of behavioral parent 
training and a brief teacher consultation to establish a Daily Report Card. Children who do not 
respond adequately to the behavioral intervention (as measured by rating scales completed by 
the child’s teacher at specified 

intervals) receive medication Figure 1. Example adaptive intervention in Childhood ADHD 
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intervention. Children who intervention intervention 
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Educators and researchers have studied or implemented adaptive interventions in various 
settings, including to prevent absenteeism (e.g., Impact Evaluation of Parent Messaging 
Strategies on Student Attendance), prevent conduct problems among high-risk children 
(Bierman, 2002), prevent substance use in children (Dishion & Kavanagh, 2000; Dishion et al., 
2002), improve reading (Connor et al., 2013; Connor et al., 2011), treat childhood ADHD 
(Pelham et al., 2016), and address the needs of children with autism spectrum disorder (ASD; 
Almirall et al., 2016). 


Researchers often have many questions that prevent them from building a high-quality 
adaptive intervention. Typically, these questions relate to the effectiveness of specific 
components that make up an adaptive intervention—that is, questions concerning building an 
adaptive intervention. They must determine which intervention options are most beneficial for 
particular students, the best way to monitor students for the purpose of adapting the 
intervention, and the best way to individualize the intervention to address the unique and 
changing needs of individuals. Often these questions cannot be answered using existing 
theories of change, expert opinion, or the literature. 


The sequential, multiple assignment, randomized trial (SMART) is a novel multistage, 
experimental design that can help education researchers address these types of scientific 
questions used to build adaptive interventions. Despite the critical role adaptive interventions 
can play in various domains of education, research using the SMART design (or other designs) 
to systematically build effective adaptive interventions in education is still in its infancy. Most 
SMART studies have been implemented in health sciences. The SMART is a relatively new 
experimental tool, hence most educational researchers are not yet exposed to this design as 
part of their formal graduate training. 


This report is for education scientists interested in developing high-quality adaptive 
interventions and understanding the usefulness of SMARTs in building them. The report will 
address how adaptive interventions can contribute to educational practice and how SMARTs 
can support educational researchers to construct adaptive interventions. We do not intend this 
report to be highly technical or comprehensive, but rather a guide to help researchers 
determine whether to consider adaptive interventions and whether the SMART is the 
appropriate experimental tool for their research interests. We encourage readers who have 
additional questions beyond what is addressed in this report to review the additional resources 
referenced in the appendixes. 


The report is organized as follows: 


e Introduction to the components of adaptive intervention. 


e Overview of SMART studies and their role in building empirically based adaptive 
interventions. 


e Typical primary and secondary research questions motivating SMART studies, including 
an overview of how investigators can use data from a SMART to address research 
questions. 

e Guidelines investigators can use to determine if a SMART design is appropriate. 

e Appendixes with (1) a glossary of key terms (Appendix A), (2) a compilation of common 
misconceptions about adaptive interventions (Appendix B) and SMART designs (Appendix 
C), and (3) links to some websites with additional resources such as sample size 
calculators or SMART data analysis software (Appendix D). 


Throughout, we use the IES-funded Childhood ADHD SMART study referenced above to 
illustrate ideas. ADHD can negatively impact a child’s ability to attend and perform at school 
(Kent et al., 2011). According to one study using nationally representative data (Xu et al., 2018), 
the estimated prevalence of diagnosed ADHD among U.S. school-aged youth (ages 4-17) in 
2016 was 10.2 percent. Further, students with ADHD incur a significant monetary cost to the 
U.S. education system (Robb et al., 2011), and ADHD is one of the most well-represented 
disorders within special education settings (Fabiano & Pyle, 2019). Evidence-based treatments 
for ADHD include medication with psychostimulants (Conners, 2002; Greenhill et al., 2002) and 
behavioral interventions (Evans, Owens, & Bunford, 2014; Fabiano et al., 2009). However, 
questions remain about the best way to sequence and individualize these treatment options to 
improve student classroom behavior and academic outcomes (Pelham et al., 2016). The 
Childhood ADHD SMART study was conducted to address this knowledge gap. Appendix E 
includes two additional examples of SMART studies in education settings to illustrate possible 
variations in SMART designs. 


Adaptive Interventions 


What Is an Adaptive Intervention? 

In an adaptive intervention, the practitioner (or some other entity) modifies the dosage 
(duration, frequency, or amount), type, or delivery mode of an intervention in order to meet 
the unique and changing needs of individuals (or organizations). Adaptive interventions can 
guide the efforts of teachers, therapists, school professionals, health service providers, or other 
education practitioners to deliver individualized interventions. An adaptive intervention is 
prespecified and clearly defined via explicit decision rules that use observations from students 
at program entry and as they progress through the intervention to guide whether, when, or 
how to modify the intervention. The prespecified nature of adaptive interventions increases 
replicability in authentic education settings and in research that aims to further optimize or 
evaluate their effectiveness. 


What Are the Components of an Adaptive Intervention? 

Consider again the example adaptive intervention (Figure 1) for improving the school 
performance of children with ADHD. At the beginning of the school year, all children receive the 
behavioral intervention; then, at week 8 (and each month thereafter), the intervention is 


individualized (i.e., different intervention options are offered to different subgroups of 
individuals) based on the child’s response status. 


Schematics such as the one shown in Figure 1 are a common and useful way of describing an 
adaptive intervention. A second way to describe an adaptive intervention is via the use of “if- 
then” statements. Figure 2 uses if-then statements to describe the same adaptive intervention 
depicted in Figure 1. This approach makes it clearer that adaptive interventions are composed 
of a sequence of decision rules. 


Figure 2. Adaptive intervention components 


Offer behavioral intervention 


Decision Points: Then, at week 8 (and every month, thereafter) 
Decision Rule: 


Decision Point: At program entry 
Decision Rule: _ First-Stage treatment = 


Intervention Options 


Tailoring Then, Second-Stage treatment = 
Variable 


Continue with behavioral intervention 


Proximal Outcome: Child's monthly attainment of individualized target behaviors during the school year 


Then, Second-Stage treatment = 


Distal Outcome: Reduction in observed classroom rule violations from start to end of the school year 


Every adaptive intervention contains the following six key components, each described below. 


Decision Points: Points in time in which the practitioner is faced with making an intervention 
decision. In the example adaptive intervention, decision points occur at program entry, at week 
8, and each month thereafter. 


Tailoring Variable(s): Information about the individual that practitioners use to decide whether 
and how to modify the intervention. In the example, the tailoring variable is the child’s 
response status, a binary measure that distinguishes each child as a responder or 
nonresponder. Note that this tailoring variable is a summary of two teacher-rated measures: 
the Impairment Rating Scale (IRS; Fabiano et al., 2006) and the Individualized List of Target 
Behaviors (ITB; Pelham et al., 1992). 


Intervention Options: Different types of treatments, tactics, dosages (duration, frequency, or 
amount), or modalities used to deliver intervention. An intervention stage refers to the time 
after a decision point during which the individual experiences the assigned intervention option. 
In the example, there is one first-stage intervention option (behavioral intervention) and two 
second-stage intervention options (augment with medication or continue with the initial 
intervention). The second-stage intervention options are tailored to the participant’s response 
to the first stage. 


Decision Rules: Prespecified intervention procedures that link the tailoring variable(s) to 
specific intervention options. For each decision point, the decision rules prespecify the 
appropriate intervention option under various conditions. In the ADHD intervention, the first 
decision rule offers behavioral intervention to all children who enter the program. Subsequent 
decision rules (monthly starting at week 8) offer an augmented intervention consisting of 
behavioral intervention and medication for children identified as nonresponders or continued 
behavioral intervention for children identified as responders. 


Distal Outcome: The ultimate, long-term goal of the intervention. The distal outcome in the 
example adaptive intervention is reduction in observed classroom rule violations from the start 
to the end of the school year. 


Proximal Outcomes: Short-term goals or mechanisms of change the intervention is intended to 
impact to achieve the distal outcome. The proximal outcomes in the example adaptive 
intervention include the child’s monthly attainment of individualized target behaviors (for 
example, work completion, complying with teacher directions, behavior toward peers) in the 
course of the school year. 


What Is the Rationale for Adaptive Interventions in Education? 

Ideally, an adaptive intervention offers the best intervention to each student and provides only 
as much intervention as necessary at any given time. This approach can help practitioners, 
schools, and policymakers more effectively allocate scarce resources to those students who 
need it most. Adaptive interventions can be useful in settings where: 


(1) Not all students benefit from the same intervention (that is, students are likely to have 
different responses); 


(2) The intervention that works now for one student may not work in the future for the 
same student or vice versa (such as when a student initially responds to an intervention 
but later stops progressing); and 

(3) Effective interventions cannot be made available to all students in the target population 
or for the entire duration of intervention (perhaps due to limited time, money, or 
other resources). 


Several organizing frameworks highlight the importance of modifying interventions to address 
the changing needs of students (Sugai & Horner, 2009). For example, a multitiered system of 
supports (MTSS; Sugai et al., 2019) is a general education service delivery model aiming to 
achieve academic and behavioral outcomes for all students. MTSS uses increasingly intensive 
tiers of instruction and intervention combined with an objective assessment system to deliver 
individualized support that address students’ academic and behavior needs (Pullen et al., 2018). 
MTSS is considered an overarching framework, or an “umbrella” to a range of tiered systems of 
support, such as response to intervention (RTI) and Schoolwide Positive Behavior Support 
(SWPBS; Sugai & Horner, 2009). RTI, which traditionally focuses on student academics 
(Goodman-Scott & Grothaus, 2017), begins with high-quality instruction for all students in the 


general student population. Practitioners closely monitor student progress to make good 
instructional and intervention decisions, and those students who show signs of academic 
difficulties receive more intense services in order to address their difficulties and improve 
academic outcomes (Berkeley et al., 2009; Fuchs & Fuchs, 2007; Sugai & Horner, 2009). SWPBS, 
which traditionally focuses on student behavior (Pullen et al., 2018), attempts to build the 
social culture necessary for schools to be effective learning environments. This approach 
includes (1) primary tier interventions—typically, screening and universal interventions; (2) 
secondary tier interventions, which typically include more structured intervention practices, 
more frequent behavior feedback, and more active supervision; and (3) tertiary interventions, 
which typically include more intensive support. A protocol specifies how evaluations of student 
responsiveness to a particular tier should determine in practice whether and when a student 
should transition from one tier to another (Sugai & Horner, 2009). 


While the example frameworks described above highlight the value of adaptive interventions in 
the context of multitier interventions, this approach could also benefit other domains. For 
example, in the area of professional development, adaptive interventions can provide a clear 
protocol for offering additional training and resources based on teachers’ changing needs for 
support. Similarly, in the area of interactive learning environments (such as computerized 
systems that engage users in learning processes [Renkl & Atkinson, 2007]), adaptive 
interventions can help preplan when and how the system or the teacher should offer additional 
support (automated, human, or both). (See Nahum-Shani et al., 2013, for examples of how 
adaptive interventions can be used to specify when and how the system should provide or 
withhold a hint when learners press the support button during a problem-solving task.) 


The frameworks described above implicitly emphasize the need to “step up” the intervention 
with more intense (and often costlier) options for those who need it (see “stepped care” 
models in substance use; Brooner & Kidorf, 2002; Sobell & Sobell, 2000). Adaptive 
interventions, however, can also be used to preplan whether and how to “step down” an 
intervention, offering less intense or less costly intervention options to those who are doing 
well. Adaptive interventions can also combine stepped up and stepped down strategies. 


How Can Adaptive Interventions be Expanded Beyond the Student-Level? 

The adaptive intervention for the childhood ADHD example operates and produces outcomes at 
the student level. However, adaptive interventions also can be designed to impact student level 
outcomes by using intervention components at other levels, such as the teacher, classroom, 
school clinic, or school. 


For example, the Adaptive School-Based Implementation of Cognitive Behavioral Therapy (CBT) 
Study (Kilbourne et al., 2018; see details in Appendix E) includes four adaptive interventions 
aiming to improve student-level outcomes by improving the quality and total number of CBT 
sessions delivered by mental health school professionals. One school-level intervention option 
employed is an ongoing, CBT-skills-coaching intervention for all mental health professionals 


within the school. Another intervention option, known as Facilitation, addresses school-level 
barriers that prevent school professionals from adopting or implementing evidence-based 
practices, such as communication barriers or conflict with school leadership. Further, the 
investigators assessed the tailoring variable (response status) at the school level, classifying 
schools as either nonresponsive or responsive based on the quality of CBT delivery within the 
school. In this example, all intervention components are employed at the school level, but the 
ultimate goal is to improve outcomes at the level of the students. 


Adaptive interventions can also include components employed at multiple levels 
simultaneously. These adaptive interventions are called multilevel adaptive interventions. For 
example, the adaptive interventions considered in the IES-funded pilot study, Getting SWART 
About Social and Academic Engagement of Elementary Aged Students With ASD (Grant # 
R324U150001 https://ies.ed.gov/funding/grantsearch/details.asp?ID=1758), includes 
intervention components delivered at multiple levels, including the school, the classroom, and 
the student (see Figure 1 of Almirall et al., 2018b). These adaptive interventions aim to improve 
social and academic outcomes for schoolchildren with ASD. One of the adaptive interventions 
considered in this study begins with a school-level intervention known as Remaking Recess 
(Kretzmann, Shih, & Kasari, 2015), which is intended to promote social engagement during 
unstructured school times on the playground. This is followed (at week 4) by a classroom-level 
intervention known as Classroom Supports (National Research Council, 2001), which provides 
the teacher with skills to improve the behavioral regulation and classroom management of 
children, and then (at week 12) by an intervention that targets the individual child via either the 
parent or the child’s peers. The tailoring variable is at the student level, whereby at week 20, 
children are identified either as responders or slow responders based on playground 
observations taken by a paraprofessional. Early responders continue with the assigned child- 
level intervention; slower responders are offered an integrated intervention that targets the 
child via parents and peers. 


In sum, there is no requirement that intervention options in an adaptive intervention be 
employed only at the student level, even when the goal is to improve student-level outcomes. 
Depending on the theory of change guiding intervention development and practical 
considerations, an adaptive intervention can employ components at multiple levels beyond the 
student. In the autism example above, intervention is provided at the level of the school, the 
level of the classroom, and then at the level of the child. Appendix B speaks to additional 
misconceptions investigators often have regarding adaptive interventions. 


How Can Intervention Components be Selected and Integrated to Form an 
Effective Adaptive Intervention? 

To construct an adaptive intervention, an investigator must select and integrate the six 
components of an adaptive intervention (decision points, tailoring variables, intervention 
options, decision rules, distal outcome, and proximal outcomes). Ideally, the investigator will 


rely on empirical evidence, existing theories, practical considerations, clinical considerations, or 
some combination of these to guide this process. 


As is typical for any type of intervention research, the investigator will begin by identifying a 
target population (for example, school children with ADHD), selecting a primary distal outcome, 
then articulating a clinically or academically meaningful goal for the target population based on 
this distal outcome (for example, reduction in observed classroom rule violations by the end of 
the school year). 


Then, the investigator will identify proximal outcomes by articulating a theory of change. This 
model specifies mechanisms of change—processes or events—that can lead to (or prevent) the 
distal outcome. These mechanisms (the proximal outcomes) should be malleable; that is, they 
should be subject to change via an intervention. In the childhood ADHD example, the 
investigators selected the proximal outcomes (child’s attainment of individualized target 
behaviors) based on evidence suggesting that they would reduce observed classroom rule 
violations by the end of the school year and that these mechanisms of change could potentially 
be modified via an intervention. 


Guided by the selected proximal outcome(s), the investigator will identify intervention options 
that are likely to impact the selected proximal outcomes in the desired direction. In the childhood 
ADHD example, behavioral intervention and medication were selected based on evidence 
suggesting that they likely improve the child’s attainment of individualized target behaviors. 


The theory of change should also guide the selection of tailoring variables. These variables 
contain information that is useful in identifying subgroups of participants who will likely benefit 
from different intervention options. Often, the mechanisms of change selected as proximal 
outcomes are also selected as tailoring variables because this information can be used to 
identify early those participants who do not experience sufficient improvement and require 
intervention modification. In the childhood ADHD example, investigators selected response 
status as a tailoring variable based on evidence indicating that responders and nonresponders 
require different subsequent intervention options: responders will do well if they continue with 
the initial intervention whereas nonresponders require intervention modification. The child’s 
attainment of individualized target behaviors is both the proximal outcome and one of the 
measures used to assess the tailoring variable (early response status). This is based on evidence 
suggesting that children who do not experience improvement in attaining an individualized 
target behavior early in the course of the intervention are less likely to show progress toward 
the proximal or distal outcomes without a change in treatment. 


A theory of change requires a determination of when meaningful change is expected to occur. 
This consideration guides the selection of decision points in an adaptive intervention. For 
example, in the childhood ADHD example it was expected that at minimum two months were 
needed following the start of intervention before expecting meaningful change in the child’s 


attainment of individualized target behaviors. Additionally, meaningful change was expected to 
occur at monthly time intervals thereafter. 


Finally, the investigator uses this theory of change to integrate the components of an adaptive 
intervention such that they work together to improve the proximal and distal outcomes. To 
clarify what it means and why it is important to select and integrate the components of an 
adaptive intervention so that they work well together, consider possible positive or negative 
synergies between the components comprising an adaptive intervention. In this context, 
positive synergies represent scenarios in which a component that seems suboptimal as a 
standalone component leads to the most beneficial outcomes as part of an adaptive 
intervention (i.e., as part of a sequence of individualized interventions). Negative synergies 
represent scenarios in which a component that seems beneficial as a standalone component 
leads to suboptimal outcomes when used as part of an adaptive intervention. Below we provide 
hypothetical examples of positive and negative synergies between first and second-stage 
intervention options and their implications on the effectiveness of the adaptive intervention. 


Positive Synergies: Consider the decision to employ behavioral intervention as the first stage 
intervention option in the adaptive intervention for childhood ADHD (Figures 1 and 2). Ideally, 
this decision is based on evidence suggesting that the behavioral intervention is an effective 
approach to initiate an adaptive intervention that then adds medication for nonresponding 
children, while continuing the behavioral intervention for responding children. For example, 
suppose that existing evidence indicates that, although the behavioral intervention leads to less 
short-term improvement in behavioral outcomes compared to medication (e.g., up to 4 
months; Swanson et al., 1993; Craig et al., 2015; Pelham et al., 1993), the behavioral 
intervention builds psychosocial skills that likely improve the effectiveness of the second-stage 
options for nonresponders (add medication) and responders (continue with the behavioral 
intervention) (Pelham et al., 2016). This implies that as a standalone component the behavioral 
intervention may not be as effective as medication in the short term. However, the behavioral 
intervention may be highly effective as part of an adaptive intervention since the second-stage 
options likely capitalize on the psychosocial gains achieved by offering the behavioral 
intervention initially. This is an example of positive synergy between the initial intervention 
option and the subsequent options. In this case, selecting to begin with the behavioral 
intervention is expected to lead to a more effective sequence of intervention options compared 
to starting with medication. 


Negative Synergies: Consider the decision to add medication to the behavioral intervention as 
the second-stage intervention option for nonresponders in the adaptive intervention for 
childhood ADHD (Figures 1 and 2). Ideally, this decision is based on evidence suggesting that 
adding medication is an effective approach to treating nonresponding children who initially 
received a behavioral intervention. Suppose, instead, that the investigator decides to add 
medication based on existing evidence suggesting that medication is effective as a standalone 
intervention (rather than as part of a sequence of individualized interventions that starts with a 


10 


behavioral intervention). This may seem like a logical choice given the extent of research on 
medication for ADHD. This approach may lead to the development of a suboptimal adaptive 
intervention if the effectiveness of medication differs depending on whether it is provided as a 
standalone intervention or following another intervention. For example, it is possible that 
although medication is an effective standalone approach to treating childhood ADHD, it is 
actually less effective when offered to nonresponders who first received a behavioral 
intervention. If this were true, then this is an example of negative synergy between the initial 
intervention option (behavioral intervention) and the subsequent option (adding medication for 
nonresponders). In this case, selecting to add medication as a second-stage option for children 
nonresponsive to a behavioral intervention would ultimately undermine the effectiveness of 
the adaptive intervention. 


Summary: Positive or negative synergies between the first-stage and second-stage intervention 
options could be sufficiently strong so as to lead to delayed effects of first-stage interventions. In 
this setting delayed effects occur, for example, when (1) a first-stage intervention is not effective 
in the short term but is effective in the longer term when followed by second-stage interventions 
as part of an adaptive intervention, or (2) when there is a reversal in the direction of the short- 
term effectiveness of a first-stage intervention vs. its effectiveness when followed by a second- 
stage intervention as part of an adaptive intervention (Nahum-Shani et al., 2019). These scenarios 
highlight the importance of selecting intervention components based on empirical evidence 
demonstrating their efficacy as part of a sequence of decision rules comprising an adaptive 
intervention, rather than as standalone components. However, often such evidence does not 
exist (Collins, 2018; Collins & Kugler, 2018), and there are unanswered scientific questions 
regarding the selection and integration of components in an adaptive intervention. In the next 
section, we discuss how SMART studies can be used to address these questions. 


Sequential Multiple Assignment Randomized Trials (SMART) 


What Is aSMART? 

The SMART (Lavori & Dawson, 2014; Murphy, 2005) is an experimental design that education 
researchers can use to efficiently answer multiple scientific questions concerning the selection 
and integration of the components that make up an adaptive intervention. A SMART involves 
multiple stages of randomizations, meaning that some or all individuals (or organizations) 
participating in a SMART are randomized more than once; this is the SMART’s defining feature. 


Examples of SMARTs outside of education. Researchers have used SMARTs to develop 
adaptive interventions for depression (Gunlicks-Stoessel et al., 2016), weight loss (Naar-King et 
al., 2016), cancer (Kidwell, 2014), smoking cessation (Fu et al., 2017), substance abuse (McKay 
et al., 2015), schizophrenia (Shortreed & Moodie, 2012), and juvenile delinquency prevention 
(August, Piehler, & Bloomquist, 2016). 


Examples of SMARTs in education. To our knowledge, the few completed SMART studies in the 
area of education mainly have addressed ASD (Kasari et al., 2014; Almirall et al., 2016; Almirall 
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et al., 2018a) and ADHD (Pelham et al., 2016). Other SMARTs currently in the field include 
the following IES-funded studies: 


e Impact Evaluation of Parent Messaging Strategies on Student Attendance (Contract # ED- 
IES-16-C-0017 https://ies.ed.gov/ncee/projects/evaluation/other_messaging.asp) 

e Cohesive Integration of Behavior Support Within a Process of Data-Based Intervention 
Intensification (Grant # R324N180018 https://ies.ed.gov/funding/grantsearch 
details.asp ?ID=2212) 

e Adaptive Response to Intervention (RTI) for Students With ADHD (Grant # R305A170523 
https://ies.ed.gov/funding/grantsearch/details.asp?ID=2073 


The latter two studies—the Cohesive Integration of Behavior Support Within a Process of Data- 
Based Intervention Intensification study and the Adaptive Response to Intervention (RTI) for 
Students with ADHD—represent examples that promise to inform adaptive interventions within 
the context of the MTSS framework. 


Ongoing SMARTs also include a study to improve the adoption of Cognitive Behavioral Therapy 
in school settings (Kilbourne et al., 2018) and a study to improve social communication 
outcomes in children with autism who are minimally verbal (Adaptive Intervention for 
Minimally Verbal Children With ASD; Pl: Kasari), reviewed briefly in Appendix E. 


What Types of Scientific Questions Can Be Answered With a SMART? 
SMARTSs are typically motivated by scientific questions concerning (1) the comparison of 
intervention options at specific decision points of an adaptive intervention, (2) the comparison 
of multiple adaptive interventions, and (3) the identification of additional tailoring variables in 
an adaptive intervention. Below, we provide examples of scientific questions consistent with 
each of these three motivations. 


Suppose in the context of the childhood ADHD example, the investigators did not have 
sufficient empirical evidence to determine the answers to the following four questions: 


Question 1: Which intervention option is the best to offer initially as part of an adaptive 
intervention: behavioral intervention or medication? 


Question 2: What subsequent intervention option is the best for children who show signs 
of early nonresponse to the intervention: is it better to augment the original intervention 
option with another type of intervention (Augment) or enhance the intensity of the initial 
intervention (Intensify)? 


Question 3: What sequence of decision rules (that is, which adaptive intervention) is best: 
the sequence described in Figures 1 and 2 or an alternative sequence that recommends 
medication initially and then augmenting with a behavioral intervention for 
nonresponders and continuing medication for responders? 
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Question 4: Should the adaptive intervention include additional tailoring variables beyond 
the child’s early response status? Specifically, can information regarding the child’s 
adherence to the initial intervention be used to identify a subgroup of nonresponding 
children who would benefit more from the Augment option than the Intensify option? 


Questions 1 and 2 both concern the comparison of intervention options at specific decision 
points of an adaptive intervention, question 3 concerns the comparison of multiple adaptive 
interventions, and question 4 concerns the identification of additional tailoring variables 
beyond those already included in an adaptive intervention. 


How Can SMART Data Be Used to Address Common Scientific Questions 
Concerning the Development of an Adaptive Intervention? 

Figure 3 depicts the ADHD SMART study, discussed earlier in this paper, to systematically 
answer scientific questions concerning the construction of an adaptive intervention for children 
aged 5 to 12 with ADHD (PI: 
Pelham). At the beginning of 
the school year, investigators 
randomly assigned 152 children 
to either low-dose medication 
(MED) or low-intensity 
behavioral intervention (Bl) 
(First stage). At week 8, and 
monthly thereafter, children 
were assessed for 
responsiveness. Children 


Figure 3. Childhood ADHD SMART study 
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the intensity/dose of the initial ® = Medication; BI = Behavioral Intervention 

intervention (Intensify: MED+ 
or BI+) or add the other intervention (Augment: MED+BI or BI+MED) (Second stage). Children 
showing adequate response (responders) continued with the initial intervention and were 
monitored monthly. The primary outcome was classroom rule violations based on objective 
observations of classroom behavior at the end of the school year. This experimental design 
resulted in six experimental conditions, labeled A through F in Figure 3, and four embedded 
adaptive interventions (described in more detail below). 


Response 


= Randomization 


Note that such a design ensures that randomized subgroups are compositionally similar given 
past characteristics (known or unknown) up to the point of randomization. For example, the 
group of children randomized to MED (conditions A+B+C) are compositionally similar to the 
group of children randomized to BI (D+E+F) in terms of all known or unknown characteristics 
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prior to the beginning of the school year (baseline characteristics). Similarly, for example, the 
group of BI nonresponders randomized to Augment (condition E) is compositionally similar to 
the group of BI nonresponders randomized to Intensify (condition F) in terms of all known or 
unknown characteristics up to the point of nonresponse. However, such a design does not 
assume that the children within each of the conditions A through F are homogeneous; for 
example, nonresponding children within condition F may differ based on the time at which they 
transitioned from first stage BI to second-stage BI+. 


Data from this childhood ADHD SMART can answer the four scientific questions outlined earlier. 


1. Which intervention option is the best to offer initially? 


This question concerns the comparison of two initial intervention options. Specifically, the 
investigator would like to determine whether starting with a behavioral intervention is better 
than starting with medication in terms of classroom behavior (primary outcome) at the end of 
the school year. The investigator can make this determination by comparing the mean outcome 
for the participants who received medication initially (conditions A + B + C) to the mean 
outcome for those who received the behavioral intervention (conditions D + E + F; see Figure 3). 


This design amounts to a two-group comparison between half of the sample who started with 
medication and the other half who started with a behavioral intervention. This comparison 
represents the “main effect” of the first-stage intervention options, averaging over the second- 
stage intervention options for responders and nonresponders. The investigator can use 
standard data analysis methods for this comparison. Specifically, he or she can use standard 
regression approaches for comparing initial intervention options in terms of an end-of-study 
outcome (Nahum-Shani et al., 2012a). The investigator can also use standard longitudinal data 
analysis methods (for example, linear mixed models [Verbeke & Molenberghs, 2009], also 
known as mixed-effect, random effects, hierarchical linear, or growth curve models) for 
comparing initial intervention options in terms of outcome trajectories. 


2. What subsequent intervention option is the best for nonresponders? 


This question concerns the comparison of the second-stage intervention options among 
nonresponders. Specifically, the investigator would like to determine whether nonresponders 
would benefit more from the Intensify option vs. the Augment option based on changes in 
classroom behavior (primary outcome) at the end of the school year. The investigator can make 
this determination by comparing the mean outcomes for the nonresponders who were 
assigned to the Augment tactic (conditions B + E) and those who were assigned to the Intensify 
tactic (conditions C + F; see Figure 3). This design amounts to a two-group comparison between 
half of the nonresponders who received the augmented intervention and the other half of 
nonresponders who received the intensified intervention. This comparison represents the 
“main effect” of the second-stage intervention options among nonresponders, averaging over 
first-stage intervention options. Similar to the comparison of initial intervention options, 
researchers can use standard data analysis approaches to compare second-stage intervention 
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options among nonresponders. However, the analysis will only include nonresponders to the 
initial intervention. 


3. What sequence of decision rules (i.e., which adaptive intervention) is better compared to 
another? 

This question concerns the comparison of two embedded adaptive interventions. The 
sequential randomizations in the childhood ADHD SMART give rise to four “embedded” 
adaptive interventions (Table 1). 


Table 1: Four Adaptive Interventions Embedded in 
One of the four embedded adaptive the ADHD SMART 


interventions—labeled “BI-MED” (#3 in 
Table 1) and represented by cells D+E—is Intervention | Stage Status Stage | Fig. 2 

illustrated in Figure 3 above. All four (1) “MED- 
embedded adaptive interventions in the BI” MED 


childhood ADHD SMART use the child’s 


response status as a tailoring variable. (2) “MED- 


MED” 


Here, the investigator would like to 
determine whether students have better (G) “BE BI 


MED” 
long-term behavioral outcomes if they start 
with behavior intervention and add 
medication later for nonresponding 
children (BI-MED adaptive intervention) or 
start with medication and add in the behavioral intervention for nonresponding children 
(MED-BI adaptive intervention). 


(4) “BI-BI” 


Nonresponder 


The investigator can make this determination by comparing the outcome among the 
participants who are consistent with the BI-MED adaptive intervention (meaning responders to 
the behavioral intervention and nonresponders who were assigned to augment with 
medication; conditions D+E in Figure 3) with the outcome among the participants who are 
consistent with the MED-BI adaptive intervention (responders to medication and 
nonresponders who were assigned to augment with behavioral intervention; conditions A+B 

in Figure 3). 


Unlike the standard regression analyses associated with the main effect aims, the regression 
analysis for comparing embedded adaptive interventions in a prototypical SMART such as the 
childhood ADHD study requires a small adjustment involving weights that are a function of the 
known randomization probabilities. Nahum-Shani and colleagues (2012a) discuss the intuition 
for this adjustment. They provide details regarding the method and example code for 
comparing adaptive interventions in terms of an end-of-study outcome. Nahum-Shani and 
colleagues (2019) describe use of repeated outcome measurements to compare embedded 
adaptive interventions. 
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4. Should the adaptive interventions include additional tailoring variables? 


This question concerns whether baseline and time-varying information collected in the course 
of the SMART can be useful in identifying additional tailoring variables beyond the child’s 
response status. While the childhood ADHD SMART uses response status as the only tailoring 
variable, other data collected during the SMART study can yield useful information to further 
tailor the adaptive intervention in the future. 


For example, the investigator may wish to examine whether the second-stage intervention 
option for nonresponders should be tailored to the child’s level of adherence to the initial 
intervention. Evidence could indicate that among nonresponders, those who were nonadherent 
to the initial intervention benefit more from augmenting the initial intervention, whereas no 
differences exist (between the two subsequent tactics) among adherent nonresponders. This 
finding can inform an adaptive intervention that not only tailors the second-stage intervention 
options based on the child’s response status, but also based on his or her adherence to the first 
stage intervention—offering an augmentation to nonresponders who did not adhere to the 
initial intervention, either augmentation or intensification to adherent nonresponders, and 
continuing the initial intervention for responders. As an alternative example, the investigator 
may wish to examine whether the second-stage intervention option for nonresponders should 
be tailored based on the time at which the child showed signs of inadequate response. 


Similarly, the investigator may wish to examine whether the first-stage intervention option 
should be tailored to baseline information, such as gender, age, treatment history, or the 
severity of a child’s initial symptoms. For example, assume the results indicate that children 
who were prescribed medication in the past benefit more from the behavioral intervention 
initially compared to medication, whereas those who were not prescribed medication in the 
past benefit more from medication initially compared to the behavioral intervention. This 
finding can inform an adaptive intervention that not only tailors the second-stage intervention 
options to the child’s response status, but also tailors the first-stage options based on whether 
the child was prescribed medication in the past. 


Data analyses to address this type of question are similar to standard moderator analyses 
(Kraemer et al., 2006). In standard moderator analyses of data from standard randomized trials, 
investigators often fit regression models with an interaction term between the (randomized) 
intervention assignment indicator and a baseline covariate. Analyses associated with the 
development of more deeply tailored adaptive interventions using data arising from a SMART 
are similar but (1) the covariates included in the interactions can be both baseline or time- 
varying, and (2) intervention assignment is time-varying (e.g., stage 1 intervention options and 
stage 2 intervention options among nonresponders). Q-learning regression is one approach that 
is easy to understand and useful for addressing this type of question (Nahum-Shani et al., 
2012b). Q-learning is a generalization of moderated regression analysis to multiple stages of 
intervention with a specific focus on exploring candidate time-varying tailoring variables. 
Accessible tutorials to this approach can be found in Nahum-Shani et al. (2012b) and Nahum- 
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Shani et al. (2017). The methodological literature concerning methods to address this type of 
scientific question is now extensive and growing rapidly. While it is outside the scope of this 
article to provide a comprehensive review, here we list a few variants or extensions of Q- 
learning which serve as a starting point (and in Appendix D we provide links to related 
software): interactive Q-learning (Laber, Linn, & Stefanski, 2014); classification based methods 
(Zhang et al., 2012; Zhang, et al., 2013); dynamic weighted ordinary least squares (Wallace & 
Moodie, 2015); and decision lists (Zhang et al., 2015; Zhang et al., 2018). 


When to Consider a SMART Design 

The starting point for considering a SMART is whether the investigator is interested in building 
an effective adaptive intervention (Collins, 2018; Collins & Kugler, 2018). If, instead, the goal is 
to evaluate an adaptive intervention that already exists, a SMART is not necessary. For example, 
if the ultimate goal is to confirm that an existing adaptive intervention is superior to a standard 
of care intervention, then a standard confirmatory randomized controlled trial (RCT) comparing 
the adaptive intervention to a suitable control may be the most appropriate design. SMART 
designs do not represent an alternative to confirmatory RCT designs. 


The SMART is a useful Figure 4. Guidelines for determining whether a SMART 
experimental tool for should be considered 

investigating how to 

select and integrate Is the intervention to 

adaptive intervention be developed an 


. Adaptive Intervention? 
components. However, it 


is not the only type of 


randomized trial design 
Is there evidence to inform the 


for building an effective 

. eu ‘ ue ; selection and integration of Por 
adaptive intervention components into an effective Design 
(Almirall et al., 2018b). Adaptive Intervention? 


Other trial designs can be 
useful depending on the 


Is there interest in answering 


scientific questions scientific questions about building 
motivating the study. This an optimized Adaptive Intervention 

: at multiple stages of the Adaptive 
section proposes three Intervention? 


questions investigators 
can ask themselves to 
help determine whether Consider a SMART 
to consider a SMART 
(see Figure 4). 


1. Is the Intervention to Be Developed an Adaptive Intervention? 
If yes, consider a SMART. Investigators may consider a SMART if the intervention they seek to 
build is an adaptive intervention. Recall that an intervention is adaptive if (1) it includes 
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individualization, namely the use of information about the targeted unit (for example, the 
student, the classroom, the school) to select intervention options, and (2) the information used 
for individualization is dynamic, meaning that the information used to decide which 
intervention option to offer can change over time, including possibly as a result of prior 
intervention. As an example, recall that in the childhood ADHD study described above, the 
investigators were motivated to develop an intervention that uses dynamic information about 
the child’s response status to decide whether to continue the initial intervention (if the child is 
responding) or modify the initial intervention (if the child is not responding adequately). 
Because the goal was to develop an adaptive intervention for childhood ADHD, the investigator 
considered a SMART. 


If no, you may not need a SMART. Investigators may consider an alternative design if the 
intervention they seek to build does not include individualization (meaning the plan is to offer 
the same intervention to all targeted 
units) or the intervention includes 
individualization solely based on 
static information (i.e., information 
that is unlikely to change as a result 
of the intervention such as age, 
gender, and stable personality 


traits). Consider a hypothetical FB 


Figure 5. Randomized trial comparing two interventions 


scenario where an investigator 
wishes to develop a nonadaptive 
intervention for childhood ADHD. In 


Beginning of End of school year 
this scenario, the investigator is school year 
interested in determining whether MED = Medication; BI = Behavioral Intervention 


this intervention should include @®) = Randomization 

either medication or behavioral 

intervention and in understanding for whom medication (vs. behavioral intervention) would be 
beneficial based on gender, age, and symptom severity. In this case, the investigator may 
consider a standard two-arm randomized trial (see Figure 5), assigning children with ADHD to 
either medication or behavioral intervention, with age, gender, and symptom severity 
measured at baseline to enable investigation of these variables as moderators of the effect of 
medication (vs. behavioral intervention). Since the goal in this hypothetical scenario is to 
develop an intervention that offers the same intervention to all children (i.e., a nonadaptive 
intervention), a SMART is not needed. 


2. Is There Evidence to Inform the Selection and Integration of Components Into an 
Effective Adaptive Intervention? 

If yes, you may not need a SMART. Investigators may consider an alternative design (like a 
standard RCT) if they are confident that they can select or construct an effective adaptive 
intervention based on existing expertise or empirical evidence. Suppose the adaptive 
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intervention for childhood ADHD in Figures 1 and 2 was built based on a combination of clinical 
expertise, results from separate prior studies suggesting the efficacy of the behavioral 
intervention and its integration with medication (though not necessarily as a stage 2 
intervention option for nonresponders to the behavioral intervention), and sufficient evidence 
supporting the feasibility of the adaptive intervention. Suppose, further, that the investigator is 
now interested in evaluating this adaptive intervention to confirm its effectiveness without 
further questions concerning how best to select and integrate its components. In this case, the 
investigator may consider a standard two-arm RCT assigning children with ADHD either to the 
adaptive intervention or to a suitable control condition (a business-as-usual intervention, for 
example) to confirm the effectiveness of the adaptive intervention (see Figure 6). Since in this 
hypothetical example investigators can build on existing evidence to select and integrate 
intervention components into an effective adaptive intervention package, a SMART is not 
needed. In such a setting, investigators should consider an RCT to confirm the efficacy of the 
adaptive intervention package compared to control. 


If no, consider a SMART. Investigators may consider a SMART if existing evidence (empirical, 
theoretical, or practical) is insufficient to guide the selection and integration of components 

into an effective adaptive intervention. The childhood ADHD SMART study was motivated by 
insufficient evidence to inform 


the selection and integration of Figure 6: Confirmatory randomized control trial comparing 
intervention components. It was an adaptive intervention to a suitable control 


unclear which intervention option 
would be best to offer initially Adaptive Intervention 


(behavioral intervention or ie contie(s) | 
Inu 

medication) and what 

subsequent intervention option 

would be best for children who Non Response’ ii 

do not respond to the 

intervention or enhance the 


intensity of the initial 

intervention). Since there were Beginning of Monthly, End of school year 
Scams , school year Beginning Week 8 

unanswered scientific questions 


concerning the construction of an MED = Medication; BI = Behavioral Intervention 
effective adaptive intervention, a = Randomization 
SMART was considered. 


3. Is There Interest in Answering Scientific Questions About Multiple Stages of the 
Adaptive Intervention? 

If yes, consider a SMART. Investigators may consider a SMART if there are scientific questions 
about multiple stages in the adaptive intervention. As noted earlier, an intervention stage 
refers to a period of time following a decision point in which the individual experiences the 
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assigned intervention option. Recall that the goal of the childhood ADHD study was to develop 
a two-stage adaptive intervention: the first stage starts at the beginning of the school year and 
ends either when the child is clarified as a nonresponder or at the end of the school year (for 
responsive children), and the second stage starts when the participant is identified as a 
nonresponder and ends at the end of the school year. This study was motivated by scientific 
questions relating to both the first stage (which intervention option is better to offer initially: 
behavioral intervention or medication) and the second stage intervention options (what 
subsequent intervention option is best for nonresponding children: add another type of 
intervention or enhance the intensity of the initial intervention). The investigators considered a 
SMART since they were interested in answering scientific questions about the selection of 
intervention options at multiple stages in an adaptive intervention. 


If no, you may not need a SMART. Investigators may consider alternative designs if the 
questions motivating the study concern only one stage in an adaptive intervention. Almirall and 
colleagues (2018) discuss how investigators can use singly randomized trials (SRTs) where units 
are randomized only once in the course of the trial, to inform the development of empirically 
based adaptive interventions when the scientific questions motivating the trial concern only 
one intervention stage or the comparison of two adaptive interventions. 


Consider a scenario where an Figure 7: Randomized trial comparing two adaptive 
investigator is researching the interventions 

following scientific question: Is the 

adaptive intervention that starts Adaptive Intervention BI-MED 

with a behavioral intervention and oo Continue (BI) 


then adds medication for 


nonresponding children and ap 


continues the behavioral R) 
intervention for responding Adaptive Intervention MED-BI 
children (BI-MED) better than Response 


- : ‘ Continue (MED) 
another adaptive intervention that 


is identical in all respects except ay 
that it begins with medication : Siimaiagan 


instead of the behavioral 

intervention (MED-BI)? Such a Beginning of Monthly, End of school year 
question contrasts two adaptive school year Beginning Week 8 

interventions. In this case, the MED = Medication; Bl = Behavioral Intervention 


(®) = Randomization 


investigator may consider a two- 
arm randomized trial that assigns 
children with ADHD to either the BI-MED adaptive intervention or the MED-BI adaptive 
intervention (see Figure 7). The trial in Figure 7 is not a SMART because participants are 
randomized only once; it is a singly randomized trial. 
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Consider a second scenario where an investigator can build on existing evidence to select 
behavioral intervention as the first-stage for childhood ADHD, but there is insufficient evidence 
to determine (1) whether augmenting with medication is better than intensifying the 
behavioral intervention for nonresponders, and (2) whether, among responders to the 
behavioral intervention, it is better to continue with the behavioral intervention or reduce its 


intensity. In this case, the 
investigator may again consider 
a singly randomized trial where 
all children with ADHD initially 
receive a behavioral 
intervention and 
nonresponders are randomized 
to two subsequent “rescue” 
options and responders are 
randomized to two subsequent 
“maintenance” options (see 
Figure 8). Again, the trial shown 
in Figure 8 is not a SMART 
because no participants are 
randomized more than once; it 
is a singly randomized trial. 


Figure 8. Singly randomized trial comparing second-stage 
options among responders and nonresponders 
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Although the example trial designs in Figures 7 and 8 are not SMARTs, both are useful for 
informing scientific questions concerning how to build an effective adaptive intervention. 
Specifically, these hypothetical trials are designed to answer scientific questions that concern 
one stage in an adaptive intervention. Hence, participants are randomized only once in the 


course of the trial. 


Summary: While all SMARTs require that an investigator be interested in answering scientific 
questions related to an adaptive intervention, not all research related to an adaptive 
intervention requires a SMART. Research relating to the evaluation of an adaptive intervention 
may require a standard RCT, and research relating to only one of the stages in an adaptive 
intervention may require a singly randomized trial. In both cases, the adaptive intervention is 
the focus of research, but since the scientific questions do not concern the selection and 
integration of components at multiple stages of the adaptive intervention, a SMART should not 
be considered. As with any study, investigators should choose the randomized trial design 
based primarily on the scientific questions motivating the investigation, as opposed to choosing 
questions that fit a particular randomized trial design. Appendix C speaks to additional 
misconceptions investigators often have regarding SMART designs. 
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Future Directions 

Because educational practice is most often conducted at multiple levels (for example, children 
nested within classrooms that are nested within schools), a critically important direction for 
future research relates to Multi-level Adaptive Interventions—that is, adaptive interventions 
that guide sequential, intervention decision-making across multiple levels. Such multi-level 
adaptive interventions include those in which intervention is first tailored at the level of the 
classroom and subsequently tailored at the level of the children within the classroom 
(depending on how each child responds to the classroom-level intervention). Multi-level 
adaptive interventions also include those in which new clusters are generated as part of an 
earlier intervention stage (e.g., a child is first offered a group therapy or online support 
network), and subsequent intervention is tailored at the level of the cluster (e.g., depending on 
how each group responds to therapy); here, the clusters are not preexisting, such as with 
classrooms or schools (Nahum-Shani et al., 2017; Nahum-Shani & Dziak, 2018). 


Directions for future research include understanding and addressing challenges that might arise 
in the conduct of multi-level adaptive interventions, as well as in the design of studies to 
optimize them. Pilot studies are one approach to investigating feasibility and acceptability 
concerns prior to mounting a full-scale trial to optimize or evaluate a multilevel adaptive 
intervention (e.g., see Schoenfelder et al., 2019; Chronis-Tuscano et al., 2016; August et al., 
2016; Gunlicks-Stoessel et al., 2016; and Almirall et al., 2012). These pilot studies may prove 
useful for understanding the challenges or limitations that may arise when employing multilevel 
adaptive interventions in authentic school settings. Future research should focus on making 
singly- or sequentially-randomized trials useful for optimizing various types of multilevel 
adaptive interventions, whether they involve preexisting clusters (Almirall et al., 2018a; Almirall 
et al., 2018b) or clusters that are generated as part of the intervention (Nahum-Shani et al., 
2017; Nahum-Shani & Dziak, 2018). 
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Appendixes 
Appendix A. Glossary of Key Terms 


This glossary defines key terms selected based on their appearance in the report and in the 
literature related to adaptive interventions and SMART designs. 


Adaptive Intervention 
An intervention design using tailoring variables to guide whether, when, or how to 
modify the intervention over time. When an adaptive intervention is delivered to an 
individual, the individual progresses through multiple intervention stages. Each stage 
begins with a decision point, and—at each decision point—tailoring variables are used 
by a decision rule to select the appropriate intervention option for the individual. An 
adaptive intervention includes decision points, intervention stages, tailoring variables, 
intervention options, and decision rules. Adaptive interventions typically do not involve 
randomization. 


Decision Rules 
Guidelines describing how information about the individual should be linked to 
intervention options. The decision rules pre-specify, for each decision point, what 
intervention option should be offered under various conditions. 


Distal Outcome 
The prespecified, long-term goal of the intervention. 


Dynamic Treatment Regimen 
This is another term for “adaptive intervention.” This term is more commonly used in 


the statistical, epidemiological or clinical trials literature. Other terms that have been 
used in place of “adaptive intervention” include “adaptive treatment strategy,” 
“treatment policy,” “treatment protocol,” and “treatment algorithm.” 


Embedded Adaptive Intervention 
A prespecified adaptive intervention that is part of the SMART (i.e., embedded in the 
SMART), by design. Most SMARTs include a number of embedded adaptive 
interventions. These adaptive interventions are formalized ahead of the trial. A common 
research question motivating a SMART concerns the comparison of embedded adaptive 
interventions. 


Embedded Tailoring Variable 
A tailoring variable that is used as part of the design of a SMART to restrict the provision 
of subsequent stage intervention options. 
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Enhanced Nonresponder Trial 
Like a nonresponder trial (see below), a design in which first-stage intervention is 


provided, and nonresponders are randomized to two or more intervention arms. 
However, unlike typical nonresponder trials, in an enhanced nonresponder trial, 
participants are consented to be part of the study prior to first-stage treatment, and 
both responders and nonresponders are followed for outcome assessments. An 
enhanced nonresponder trial may generate knowledge that is useful for the 
development of an adaptive intervention, but it is not a SMART because 
participants experience at most a single randomization. See Almirall et al. (2018b) 
for additional discussion. 


Intervention Decision Point 
The point in time during which an intervention decision is made. 


Intervention Options 
Alternatives for the type/dose of intervention that may be offered at any given 


intervention decision point. 


Intervention Stage 
A period of time following a decision point in which the individual experiences the 


assigned intervention option. 


Non-responder Trial 
One in which a first-stage intervention is provided, and nonresponders are randomized 


to two or more intervention arms. Typically, in a nonresponder trial, researchers do not 
measure outcomes for responders; only the randomized nonresponders are followed 
for outcome assessments. Further, typically in a nonresponder trial, participants are 
consented at or following the point of nonresponse. A nonresponder trial may generate 
knowledge that is useful for the development of an adaptive intervention, but it is not a 
SMART because participants experience at most a single randomization. 


Prototypical SMART 
A SMART where all individuals in the target population are randomized to two first- 


stage intervention options and then nonresponders to first-stage intervention are 
rerandomized to two second-stage intervention options. This is currently the most 
common type of SMART. Not all SMARTs will employ a prototypical SMART design; the 
type of SMART designed will depend on the scientific questions the investigator seeks to 
answer. 


Proximal Outcomes 
The prespecified, short-term goals the intervention is intended to impact in order to 
achieve the distal outcome. The proximal outcomes guide the selection of the decision 
points, tailoring variables, intervention options, and decision rules. 
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Sequential Multiple Assignment Randomized Trial (SMART) 
An experimental design involving multiple stages of randomization, meaning that some 
or all individuals (or organizations) participating in a SMART are randomized more than 
once. In a SMART, the sequential randomizations can be generated upfront (e.g., prior 
to the beginning of first-stage treatment), rather than sequentially, but revelated in real- 
time. However, due to anticipatory effects, generating the randomized lists ahead of 
time (and attempting to keep them locked) could inadvertently influence outcomes 
prior to subsequent stages should research staff, intervention staff, and/or participants 
have knowledge of subsequent stage treatment assignments. Hence, a recommended 
practice is to conduct the randomization and reveal the assigned options in “real time,” 
namely at their designated decision points (see Nahum-Shani et al., 2019). 


Tailoring Variable 
Baseline or time-varying information from the individual that is useful in deciding the 
type/dose of intervention at each decision point. 


Unrestricted SMART 
The prototypical SMART is an example of a restricted SMART in that the second-stage 
intervention is restricted to nonresponders only. In an unrestricted SMART, all 


individuals are randomized repeatedly in a way that does not depend on prior 
information. 
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Appendix B. Common Misconceptions About Adaptive Intervention Designs 


Misconception #1: The same tailoring variable must be used for all individuals in an adaptive 
intervention. 


e There is no requirement that the same tailoring variable be used for all individuals in an 


adaptive intervention. 


e A tailoring variable (or a set of tailoring variables)—including how/when the measures 


that make up the tailoring variable are collected (e.g., monitoring schedule)—could differ 
by individual, including based on previous information about the individual. 


- Example of how the tailoring variable may differ by previous intervention: To 
determine how to intervene next, a child with autism who begins with an 
intervention that focuses on Discrete Trials Training (Smith et al., 2001) might be 
monitored based on the number of unique words used; whereas, a child who 
begins with a more naturalistic, play-based intervention such as might be 
monitored based on social engagement. 


- Example of tailoring frequency: individuals at higher risk for substance use relapse 
might be monitored more frequently for environmental risk than individuals with 
lower risk. 


Misconception #2: An adaptive intervention must recommend a single intervention component 
at each level of a tailoring variable. 


There is no requirement that an adaptive intervention recommends only a single 
intervention component at each level of a tailoring variable. 


At any one or more decision points, an adaptive intervention could recommend a set of 
interventions (components) instead of a single intervention (component). 


- For example, for certain individuals at certain decision points, there may be no 
evidence that a single intervention (component) is better than another. In this 
case, the adaptive intervention may recommend a set of interventions 
(components). 


Misconception #3: Adaptive interventions seek to replace clinical judgement. 


The goal of adaptive interventions is not to replace clinical or educational practice. 
The goal of adaptive interventions is to guide practice. 
Clinical judgement may play a role in the assessment or collection of tailoring variables. 


In cases where an adaptive intervention recommends a set of interventions 
(components), clinical judgement could be used to make the decision about which 
intervention to assign/recommend. 


- For example, consider the results of a hypothetical study which suggest that there 
is no evidence of a difference between increased behavioral intervention dose 
(+BMOD) or adding medication (+MED) among children with ADHD identified as 
nonresponders to initial behavioral modification. Based on these results (until 
additional information becomes available), an adaptive intervention might 
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recommend the set of both +*BMOD or +MED for nonresponders, and leave the 
final decision up to the child, parent, therapist, or schoolteacher. 


Misconception #4: Adaptive interventions are relevant only in clinical treatment practice. 
e Clinical treatment practice is not the only domain that could benefit from adaptive 
interventions. 


e Adaptive interventions are relevant in any domain where sequential (or dynamic) 
intervention decision-making is necessary. These include the following: 


— preventive interventions, such as those designed to reduce risky behavior; 


- education interventions, such as those targeting academic achievement or 
absenteeism; 


- health promotion interventions designed to encourage healthy habits; or 


- implementation interventions aimed at improving the uptake of evidence-based 
treatments. 


Misconception #5: Adaptive interventions involve randomization. 
e Adaptive interventions (typically) do not involve randomization(s) 
e Adaptive interventions (typically) are prespecified interventions that use clearly 


articulated decision rules to link a value of a tailoring variable to an intervention 
(component) or a set of interventions (components). 


e Randomization is used in studies that seek to develop or evaluate adaptive interventions, 
such as standard randomized controlled/clinical trials, enhanced (non)responder trials, or 
sequential multiple assignment randomized trials (SMARTs). 


33 


Appendix C. Common Misconceptions About SMART Research Designs 


Misconception #1. Sample size requirements are prohibitively large, and sample size 
calculations for a SMART design are complicated. 
e As with any trial, sample size calculations ina SMART are a function of the hypothesis 
tests (or estimation procedures) related to the primary aim of the SMART. 


e SMART designs do not necessarily require large sample sizes or complicated sample size 
calculations. 


e There are now a number of easy-to-use sample size calculators for data arising from a 
SMART, including calculators for an end-of-study continuous outcome, an end-of-study 
binary outcome, a continuous repeated measures outcome, a survival outcome, and for 
data arising from a cluster-randomized SMART. 


e See Appendix D for links to online sample size calculators. 


Misconception #2. All SMARTs require multiple-comparison adjustments. 
e SMART designs do not necessarily require multiple-comparison adjustments. 


e As with any randomized trial, multiple-comparison adjustments are required when 
addressing the primary research question involves testing multiple outcome measures, 
or multiple comparisons (e.g., two or more pairwise comparisons). SMARTs are no 
different in this respect. 


Misconception #3. All research on adaptive interventions requires a SMART. 
e Not all research on adaptive interventions requires a SMART design. 


- For example, a standard two-arm randomized trial can be used to evaluate an 
adaptive intervention, by comparing it to a suitable control condition. 


e SMARTs are just one type of research design (experimental design) that researchers can 
use to build (rather than evaluate) adaptive interventions. Other designs are possible. For 
example, researchers could use an enhanced nonresponder trial, which is not a SMART, 
(Almirall et al. 2018b) to inform the development of adaptive interventions by examining 
how best to treat nonresponders to an initial intervention. 


Misconception #4. All SMARTs must include an embedded tailoring variable. 
e SMARTs do not have to include an embedded tailoring variable. 


e Anunrestricted SMART design is often used in situations when there is no scientific, 
practical, or ethical rationale for embedding a tailoring variable as part of the SMART. 


— For example, researchers may use an unrestricted SMART design to (1) randomize 
all participants to two intervention options at stage 1 and (2) rerandomize all 
participants to two intervention options at stage 2. In this example, the choice of 
stage 2 intervention is not dependent on characteristics of the participant, the 
choice of assigned stage 1 intervention option, or any outcomes to stage 1 
intervention. 
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Misconception #5. All aspects of an adaptive intervention must be randomized in a SMART. 
e Not all aspects of an adaptive intervention require randomization in a SMART (or in any 
study of adaptive interventions). 


e As with any randomized trial, a single SMART study might answer only a handful of 
questions related to the development or evaluation of an adaptive intervention. 


- For example, consider a SMART that examines how best to intervene with 
children with autism who are slow-responders to a naturalistic intervention 
without a device (by randomizing these children to two intervention options) but 
does not randomize children who are slow-responders to the naturalistic 
intervention with a device (these children all received intensified intervention). 


Misconception #6. SMARTs never include a control group. 
e ASMART may be designed to include a suitable control group. 


- For example, one of the adaptive interventions embedded in a SMART might 
represent “business as usual” which would serve as a suitable control. 


Misconception #7. SMARTs require multiple consents. 
e SMARTs differ from standard trial designs in that a participant may be randomized 
multiple times. Multiple randomizations, however, do not mean that a separate consent 
is required at each randomization point. 


e As with any randomized trial, we recommend SMARTs employ a single study consent 
protocol with all participants prior to the initial randomization. We view consent as a 
study-specific procedure; not a randomization-specific procedure. Of course, the study 
consent protocol would include informing participants of all possible sequences of 
intervention components the participant might be assigned to during the course of the 
study. 


— For example, consider a SMART that (1) randomizes children with ADHD to initial 
behavioral modification (BMOD) vs initial medication (MED), (2) continues initial 
intervention for children who are not identified as nonresponders, and (3) 
rerandomizes nonresponding children to increased intervention vs. combined 
BMOD+MED. In this study, parents/children consent once, up front (i.e., prior to 
the first randomization) to being part of the entire study (both stages of 
intervention) and to the possibility of receiving one of the four embedded 
adaptive interventions. 


Misconception #8. SMARTs are susceptible to high levels of participant attrition. 
e SMARTs are no more likely to produce high levels of participant attrition than standard 
randomized trial designs (Moodie, Karran, & Shortreed, 2016). 


Misconception #9. All intervention options randomized in a SMART must be evidence-based. 
e SMARTs do not require that all intervention options to which individuals are randomized 
be evidence based (i.e., shown to be efficacious in prior efficacy research), although in 
practice this is sometimes the case. 
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e In many cases, investigators may be interested in examining the effect of intervention 
options that are critical to building a high-quality adaptive intervention but that would 
never be studied/evaluated on their own. 

- For example, consider a SMART that randomizes individuals to different options 
for monitoring the individual for response vs nonresponse (and then also 
subsequently rerandomized responders or nonresponders). It would be unlikely 
that previous trials have examined what is the best way to monitor individuals for 


response vs nonresponse in isolation. 
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Appendix D. Useful Links 
Recommended List of Introductory Readings for Adaptive Interventions and SMART 
https://methodology.psu.edu/ra/adap-inter/bib 


List of Known SMART Projects (with Schematics) up to 2016 


https://methodology.psu.edu/ra/adap-inter/projects 


Sample Size Calculator for SMARTs with Binary or Continuous Outcome 
https://nseewald1.shinyapps.io/SMARTsize 


Incomplete List of Additional Software to Analyze Data Arising from SMARTs 
Software suite, including SAS PROC QLEARN and qlaci for R: 
https://methodology.psu.edu/downloads 


DTRreg for R: 
https://www.jstatsoft.org/article/view/vO80i02 


qLearn for R: 
https://cran.r-project.org/web/packages/qLearn/qLearn.pdf 


DynTxRegime for R: 
https://cran.r-project.org/web/packages/DynTxRegime/DynTxRegime. pdf 


iqLearn for R: 
https://cran.r-project.org/web/packages/iqLearn/iqLearn.pdf 


Decision Lists for R: 
https://cran.r-project.org/web/packages/listdtr/listdtr.pdf 
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Appendix E. Other Example SMART Studies in Education 

In the main body of this report, the Childhood ADHD SMART (Study 1) was used to illustrate 
ideas. In this Appendix, we describe two additional example SMARTs in education (Studies 2-3) 
that differ in their design features from the childhood ADHD study, which we refer to here as 
Study 1. Our goal is not to provide a detailed account of each study. Rather, our goal is to 
illustrate variety in design possibilities. For each SMART, we provide the study’s overarching 
goal, discuss the intervention options considered, discuss important design considerations, list 
the specific questions, and describe the study flow. We also summarize how Studies 1, 2, and 3 
differ from each other. 


Study 2 aims to develop an adaptive intervention for minimally verbal children with autism 
spectrum disorders (ASD) and Study 3 focuses on developing an adaptive school-based 
implementation of cognitive behavioral therapy (CBT). 


Study 2: Adaptive Interventions for Minimally Verbal Children With ASD 
Study Goal: An estimated 30- 
40 percent of school-aged 
children with ASD remain 


Study 2 differs from the childhood ADHD study (Study 1) in important 
ways: 


—- e In Study 2, both responders and slower responders are 
minimally verbal even after rerandomized at the beginning of stage 2; whereas in Study 1 only 
receiving years of nonresponders are rerandomized. 


interventions. Little is known In Study 2, the stage 2 randomization occurs at a fixed time point— 


early responders and slow responders were rerandomized at week 
6—whereas, in Study 1 the stage 2 randomization could occur at 
different time points, depending on when the child triggers a 
they are often excluded from nonresponse based on monthly ratings. 


about minimally verbal 
children with autism because 


research studies. The Study 2 includes eight embedded interventions, two of which are not 
overarching goal of this adaptive (always JASP and always DTT). In contrast, all four 
NIMH-funded study (PI: embedded interventions in Study 1 are adaptive. 

Kasari) is to build the most effective 16-week, two-stage, adaptive intervention involving four 
intervention options (explained below). 


Intervention Options: A number of intervention options are available that could help improve 
social communication in these children: Discrete trials training (DTT; Smith et al., 2001) is an 
intervention that is based on the principles of applied behavior analysis. In DTT, communication 
and related skills (e.g., sounds, gestures, or words) are taught through systematic adult 
therapist-led direct instruction. Joint attention symbolic play and emotional regulation 
combined with enhanced milieu training (JASP for short; Kasari et al., 2014) is a 
developmentally anchored behavioral intervention which assumes that communication skills 
develop from social interactions. In JASP, specific social engagement strategies, symbolic 
representations, and early communication forms are modeled and naturally reinforced by adult 
therapist responses to the child. Another option is blended JASP+DTT that combines aspects of 
the two interventions in the context of a single therapy session. A final promising intervention 
option is parent training (PT; Kaiser, Hancock, & Trent, 2007), in which the child’s primary 
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therapist coaches the child’s parent (or guardian) in skills related to JASP or DTT with the goal of 
promoting improvement in social communication in the home environment. 


Intervention Design Considerations: Both JASP and DTT are evidence based, with core principles 
of DTT generally considered to be the “standard of care.” Except for PT, which takes place in the 


child’s home, all interventions in this study take place before or after hours at the child’s 
elementary school. Due to the cost of delivering PT in the home and its conjectured role as an 
intervention designed to “generalize” intervention gains beyond the school, PT is considered 
feasible only as a second-stage intervention for children showing early signs of response 
(defined below). The blended intervention option JASP+DTT is also costly since its delivery 
requires a highly skilled therapist trained in principles of both JASP and DTT. Given JASP+DTT’s 
cost and its conjectured role as an intervention designed to accelerate improvements in 
children who are not responding quickly enough (if at all), JASP+DTT is considered feasible only 
as a second-stage intervention for children showing slow response (defined below). 


Specific Questions: The study was motivated by the following questions: (1) Is it better to begin 
intervention with JASP vs. DTT (primary question)? (2) For children who show early signs of 
response to the intervention, is it better to add parent training or continue the initial intervention? 
(3) For children who show early signs of slow response, is it better to offer combined JASP+DTT or 
continue with the initial intervention? The primary outcome is a blinded measure of change in 
spontaneous socially communicative utterances from baseline to week 16. 


SMART Design: Figure 9 


shows a schematic of Study 2. 


At baseline, children are 
randomized (with equal 
probability) to either JASP or 
DTT as first-stage 
intervention. After 
approximately 6 weeks of 
stage 1 intervention, early 
response versus slow 
response status is measured 
using the therapist-rated, 
seven-item Clinical Global 
Impressions-lmprovement 
scale (CGI-l => 3 to 7 are 
considered slower 
responders; Guy, 1976) 
adapted for use with children 
with ASD. In stage 2, early 
responders to each 


Figure 9: Adaptive intervention for minimally verbal school 
children with ASD SMART study 
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intervention are randomized (with equal probability) to continue their initially assigned 
intervention versus augmented intervention that involves PT. In stage 2, slower responders to 
each intervention are randomized (with equal probability) to continue their initially assigned 
intervention versus an augmented intervention that combines JASP and DTT. 


Study 3: Adaptive School-Based Implementation of CBT 

Study Goal: Depressive and anxiety disorders affect 20-30 percent of school-age youth. 
Cognitive behavioral therapy (CBT; Ginsburg et al., 2008) can improve outcomes in these 
disorders and ultimately lead to improved school performance. Students access mental health 
services at schools more than anywhere else, but school professionals (SPs; e.g., school 
counselors, school social workers) are generally not trained to deliver CBT, or they experience 
other barriers that limit providing effective CBT to students. The overarching goal of this NIMH- 
funded study (PI: Kilbourne) is to build the most effective 15-month, 3-stage, school-level 
adaptive implementation intervention involving three distinct implementation strategies 
(known as Replicating Effective Programs (REP; Kilbourne et al., 2007), Coaching, and 
Facilitation) to improve the uptake of CBT by SPs at schools. 


Intervention Options: A 
number of school-level 


Study 3 differs from Studies 1 and 2 in important ways: 


e In Studies 1 and 2, the goal was to develop individual-level adaptive 


intervention options are 
available that could help 
schools implement CBT 
and therefore improve 
mental health outcomes 
of its students in need of 
mental health services. 
They vary in terms of 
intensity, cost, and 
theoretical foundation: 


interventions to improve individual-level outcomes. In contrast, in 
Study 3 the goal is to develop a school-level adaptive intervention to 
improve the outcome of school professionals within schools. As a 
result, Study 3 is a clustered SMART, i.e., schools are the unit of 
randomization but the outcome is at the SP-level. 


In Study 3, the primary aim compares an adaptive intervention versus a 
nonadaptive intervention: namely, the costliest embedded intervention 
(offer REP+Coaching in stage 2, continue REP+Coaching in stage 3 to 
responding schools, and offer REP+Coaching+Facilitation in stage 3 to 
sub-optimally responding schools) versus the least expensive 
embedded intervention (offer REP alone at all stages). 


REP combines customized CBT packaging, didactic training, and technical support to improve 
CBT uptake by the SP. Coaching (Beidas et al., 2012) extends training via live supervision to 
improve the SP’s competence and acceptance. Facilitation (Kilbourne et al., 2015) is used to 
mentor SPs in strategic thinking to promote self-efficacy in championing the use of CBT and 
securing administrator support at the school. 


Intervention Design Considerations: Schools receive training on identifying schoolchildren with 
depressive and anxiety disorders who could benefit from CBT as part of the didactic in-person 
training component within REP. Coaching is considered feasible only after SPs at the school 
have had ample time to identify students in order to facilitate live supervision of CBT delivery. 
This consideration provided the rationale for possibly offering coaching at stage 2, which is 2 
months following the start of REP. Facilitation is not considered feasible during stage 1 or stage 
2 because it is designed to address prolonged barriers (i.e., processes that require ample time 
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to become visible), such as lack of self-efficacy in championing the use of CBT. In addition, 
facilitation is not considered feasible for schools that already were responding (defined below) 
by the end of stage 2, thus providing the rationale for possibly offering facilitation at stage 3, 
following the start of stage 1 REP and the possibility of coaching in stage 2. 


Specific Questions: The study includes two adaptive and two nonadaptive implementation 
interventions. The most intensive study intervention is an adaptive intervention that offers all 
schools (1) REP in stage 1, (2) REP+coaching in stage 2, (3) continued REP+coaching in stage 3 
for schools that do not require additional assistance following stage 2 intervention (defined 
below), and (4) REP+coaching+facilitation in stage 3 for schools that require additional 
assistance following stage 2 intervention. The least intensive study intervention is one that 
offers REP alone for 15 months. The study was motivated by the following questions: (1) What 
is the effect of the most intensive adaptive implementation intervention versus offering REP 
alone (primary question)? (2) Does the effect of REP+coaching vs. REP alone in stage 2 differ 
depending on school-aggregated SP factors pertaining to perceptions of CBT or prior training in 
CBT? (3) Does the effect of augmenting with facilitation vs. not in stage 3 differ depending on 
satisfaction with stage 2 implementation support or CBT delivery during stage 2? The primary 
outcome is the total number of CBT sessions delivery by SPs over the course of 18 months. 


SMART Design: Figure 10 shows a schematic of Study 3. A total of 114 high schools (including 
more than 200 SPs) received REP for 2 months in November to December 2018 (stage 1). After 
2 months, schools were randomized with equal probability to continue REP or to augment REP 
with coaching (stage 2). At the end of 2 additional months, suboptimally responding schools 
detinedasat ae ie: Figure 10. Adaptive school-based implementation of 

the school delivered CBT to CBT SMART study 

10 or fewer students) are 
rerandomized to continue 

with their previously 

assigned implementation 

strategy or to augment with 

facilitation (stage 3). Stage 3 “REP \(R) 
intervention continues for a 

total of 11 months. The total 

duration of intervention for 

all schools is 15 months. 
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